School of Education,
Social Work and Community Education, University of Dundee
Evaluation of the
efficacy of the Tweenees lessons
Ian Barron
Teaching Fellow
Child Care and
Protection
School of Education,
Social Work and Community Education, University of Dundee
School-Based Abuse
Prevention Programme Evaluation
Traditional Literature Review: 1990 -
2007
- 22 Studies (3 secondary) evaluated
school-based abuse prevention programmes over a 17 year period. The following
outcomes were identified:
- Children displayed high levels of
‘prior knowledge’
- On average small gains in knowledge
and behavioural intention (telling) were achieved
- Cognitively able middle class girls
made the most gains
- There were small increases in the
number of disclosures (8 studies)
- At least 4 lessons, boosted annually
were necessary to sustain knowledge gains
- There was no evidence that
school-based abuse prevention programmes cause any harm
School of Education,
Social Work and Community Education, University of Dundee
Meta-analyses (1990 –
2007)
Outcomes
- 4 identified meta-analyses (most
recent Davis and Gidycz, 2000) with effect sizes ranging from d = 0.47 to 1.07
moderate to high effect size)
Moderator variables
- Duration (4 lessons and booster)
and content of the programme (modelling and skill rehearsal)
- Age (older pupils achieved
greater gains)
- SES (pupils from middle class
families achieved greater gains than working class)
School of Education,
Social Work and Community Education, University of Dundee
Gaps in the research to
date
- Participants (survivors, upper
primary and secondary aged pupils’ response to programmes)
- Lack of systematic recording
(demographic, attrition rates, disclosures, effect sizes)
- Assessment measures (limited
standardised measures, over reliance on questionnaires, children’s views not
sought, children’s negative experiences not asked for, no cost-effectiveness
measures, virtually no programme integrity measures)
School of Education,
Social Work and Community Education, University of Dundee
Current
study
- Evaluation of a local school-based
abuse prevention programme – the Tweenees (VIP package)
- Design: Pre and post-test measures
with waiting list control group, matched for numbers
- School and year group sampling –
purposeful
- Sampling for interviews - stratified
sampling for age and gender and random sampling for SES
- Bias towards urban, working class,
Caucasian pupils
School of Education,
Social Work and Community Education, University of Dundee
School-Based Abuse
Prevention Evaluation: The Local Study
- 4 lessons were delivered from the
Tweenees programme over a 4 week period to a small group of survivors, as well
as primary and secondary aged pupils. All schools had volunteered to take part
of the evaluation.
- Survivors’ Group: Intervention group
– 18u delivered lessons (10 pupils) and waiting list control group (10
pupils), aged 6 to 13.
- Primary Group (Primary 7):
18u delivered lessons (68 pupils), class teacher delivered
lessons (20 pupils) & waiting list control group (59 pupils)
- Secondary Group: S1 (118) and S2
(114) including waiting list control groups for each year. Equivalent numbers
in each condition. Guidance teachers delivered the lessons.
School of Education,
Social Work and Community Education, University of Dundee
School-Based Abuse
Prevention Evaluation:
Local Study Outcome
Measures
- Knowledge & behavioural
intentions (CSKS-Q adapted, psychometric properties, used in 3 previous
studies)
- Perception of risk assessment
(perception of harm and capacity to keep self safe; exploratory – signs at
home, in school and in the community)
- Emotional impact (explicit
v’s implicit, positive/negative)
- Disclosure rates (within and
beyond lessons - generalisation)
- Customer satisfaction from
teachers and pupils
- Implementation integrity of
class lessons (exploratory)
- Tools: Questionnaires, interviews,
disclosure forms and video observation
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Survivors’ Group: Knowledge and Skills
Questionnaire Pre and post-test ‘Total Scores’ (114 total
score)
- Intervention group - significant
difference on ‘post-test total scores’ on knowledge/skills questionnaire found
(t = -4.035, p = 0.003) compared to the control group (t = 0.079, p = .939),
- 6.7 points v’s 0.1 points, SD =
5.25
- Survivors who received the Tweenees
lessons demonstrated on average, significant small gains in knowledge and
behavioural intention to tell compared to survivors who had not received the
lessons.
- Effect size d = 1.00 (high)
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Survivors’ Group: Knowledge and Skills
Questionnaire - ‘Specific Questions’
- Significant difference found on
question 10 (t = -2.862, p < .05) and 11 (t = -4.000, p < .01) post-test
scores compared to control group (t = 1.861, p = .096 & t = .557, p =
.591) respectively
- Question 10: What if your uncle was
giving you a big tight hug and you don’t like it. What would you do?
- Question 11: What if someone touches
you in a way that makes you feel uncomfortable inside and you ask them to stop
but they don’t. What would you do?
- Survivors who received the Tweenees
lessons were more likely on average to suggest more effective strategies to
these questions compared to survivors who didn’t receive the lessons, e.g.
‘saying no, getting away and telling’.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Survivors’ Group: Knowledge and Skills
Questionnaire Pre and post-test ‘Moderating Factors’
- When comparing the scores of male
and female survivors, who had received the Tweenees lessons, a significant
difference was found for Gender on question 21 [F (1,8) = 7.273, p < .05]
and question 24 [F(1,8) = 6.316, p < .05]. Given the control group was all
female there was no control group comparison.
- Female survivors who received the
Tweenees lessons were more likely on average to report higher overall ‘sense
of safety’ and ‘feeling good about themselves’ scores compared to the male
survivors who received the lessons.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Primary Evaluation (P7): Knowledge and Skills
Questionnaire - Pre and post-test ‘Total
Scores’
- 18u worker delivered lessons
- significant difference found (t = -5.433, p < .01), 3.4 points, SD
= 5.13, effect size d = 0.60
- Class Teacher delivered
lessons – significant difference found (t = -3.107, p < .01), 4.1
points, SD = 5.90, effect size d = 0.65
- Waiting list control – no
significant difference (t = 0.692, p = .492), -0.5 points, SD = 5.45
- Primary 7 pupils who received the
Tweenees lessons, whether delivered by 18u workers or their own class teacher,
achieved significantly more knowledge and behavioural intention to tell gains
on average than pupils who had not received the lessons.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Primary Evaluation (P7): Knowledge and Skills
Questionnaire - Pre and post-test ‘Specific Questions’
- 18u worker delivered lessons
- significant results found for:
Question 10: Intervention group (t =
-2.065, p < .05) compared to control group (t = 0.435, p =
0.665).
Question 14 ‘What if a friend of your
parent/carer is hugging and kissing you and then asks you to keep it a secret.
What do you do?’ Intervention group (t = -2.031, p < .05) compared to control
group (t = -0.195, p = 0.845).
- Class Teacher delivered
lessons - significant result found:
Question 13 ‘What if a 17yr old is
asking you to do something you know you shouldn’t do. You say No! I’m going to
tell. Then he says if you do I’m going to beat you up. What would you do?’
Intervention group (t = -2.486, p < .05) compared to control group (t =
-0.286, p = 0.776).
- On average primary 7 pupils who
received the Tweenees lessons were more able to suggest more effective
solutions (telling and not keeping the secret) to these specific scenarios
than pupils who did not receive the lessons.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Primary Evaluation (P7): Knowledge and Skills
Questionnaire - Pre and post-test ‘Moderating Factors’
- On pre and post-test scores, no
apparent significant differences were found between the pupils who received
the Tweenees lessons and those who did not on the moderating factors of age,
ethnicity, socio-economic status, and gender
- Age (within the P7 year group),
gender, socio-economic status and ethnicity (too small a sample) did not
appear to be factors which effected the children’s response to the Tweenees
lessons.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Secondary Evaluation: Knowledge and Skills
Questionnaire - Pre and post-test ‘Total
Scores’
- S1: Intervention group -
significant difference found (t = - 2.248, p < .05) 1.6 pts,
compared to control group (t = 1.547, p = 0.127) 0.1pts
- S2: Intervention group -
significant difference found (t = - 2.703, p < .01) 2.2pts, compared
to control group (t = - 1.594, p = .117) 1.3pts
- S1 and S2: Intervention group
- significant difference found (t = - 3.523, p < .01) 1.8pts, compared to
control group (t = - 0.393, p = .695) 0.2pts.
- In conclusion secondary S1 and S2
pupils who received the Tweenees lessons made small average knowledge and
behavioural intention gains to tell, compared to same age pupils who did not
receive the Tweenees lessons.
- The S2 group received an abuse
prevention lesson a month prior to the 4 lessons. It is unclear whether this
made any impact on the S2 results.
School of Education,
Social Work and Community Education, University of Dundee
Secondary Evaluation:
Knowledge and Skills Questionnaire - Pre and post-test ‘Specific
Questions’
- No significant results were found
for specific questions in either S1 or S2 for pupils who received the Tweenees
lessons and those in the waiting list control
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Secondary Evaluation (S1): Knowledge and Skills
Questionnaire Pre and post-test ‘Moderating Factors’
- A significant result was found for
the moderating factor of ‘Presenter’ [F(2,57) = 3.182, p < .05] from the
post-test Total Scores.
- Significant results were found for
three specific questions for Presenter
- Q 3 ‘When you are on your own who is
taking care of you?’;
- Q15 ‘Do you have to do everything
your baby sitter tell you to do?’
- Q17 ‘What if a 17yr old is asking
you to do something you shouldn’t do. You say No, I’m going to tell my
parent/carer. Then he says if you do I’m going to beat you up. What do you
do?’).
- Findings suggested that the extent
of knowledge and skill gains for S1 pupils may be partly dependant upon the
experience of the teacher presenting the lessons.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Secondary Evaluation (S2): Knowledge and Skills
Questionnaire - Pre and post-test ‘Moderating Factors’
- No apparent significant results were
found for the moderating factors of age, gender, socio-economic status,
ethnicity and presenter for S2 pupils.
- Age ‘within the S2 year group’,
gender, socio-economic status and ethnicity did not appear to be factors which
impacted on pupils response to the Tweenees lessons.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Secondary Evaluation (S1 & S2): Knowledge
and Skills Questionnaire - Pre and post-test ‘Moderating
Factors’
- No apparent significant results were
found in the whole S1/S2 group for the moderating factors of:
gender, socio-economic status,
ethnicity, presenter and year group.
- The above factors did not appear to
significantly effect pupils responses in S1 and S2 (when taken as a whole
group) to the Tweenees lessons
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Survivor Group Disclosures over 4
lessons
- Intervention Group: 12
Disclosures*
2 Child Sexual Abuse (previously
known)
8 Bullying
2 Physical Assault
*All disclosures in the lessons were
verified from video analysis of the sessions and from adults/peers within the
situations in which the harm
occurred.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Primary
Group Disclosures during the 4 lessons
- 18u worker delivered lessons
(68 pupils): 65 Disclosures
15 - physical abuse 2 - child sexual
abuse 1 – sexual assault
14 – bullying 2 – grooming 1 -
erotica
11 - physical assault 1 - attempted
stealing 1 - abduction
10 - domestic violence 1 - emotional
abuse
5 - attempted abduction 1 –
rape
- Class Teacher delivered
lessons (20 pupils): 4 Disclosures
2 attempted assault
1 domestic violence
1 domestic violence/attempted
assault
- Waiting List Control (59
pupils): 0 Disclosures
- Taking into account the numbers of
pupils in each group, there appears to be a significant difference between the
number of disclosures received by 18u workers compared to the P7 class
teacher. A number of disclosures could refer to the same incident.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Secondary Group: S1 and S2 - Disclosures during the 4
lessons
- S1 Intervention Group: 5
Disclosures*
1 domestic violence
1 bullying
1 physical assault
1 physical & emotional
abuse
1 abduction
- S1: Waiting List Control: 0
- S2 Waiting List Control: 0
*S1 disclosures occurred in the
1st session of one class, the only secondary lesson led by an 18u
worker. All disclosures were verified as having been said via video
analysis.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes:
Generalisation
Primary and Secondary Evaluations: Telephone
calls to survivors helpline (Oct – April) by school
area*
School area 2004/5 2005/6
Primary 7: 18u delivered schools 0
44
Primary 7: Class Teacher delivered
school 0 0
Primary 7: Waiting List Control schools
0 0
Primary 7: Other area schools 0
28
Secondary S1/2 : Guidance Teacher
delivered school 2 5
Secondary S1/2: Waiting List Control
school 0 0
Of the 77 calls:
36 to check the number out 4 reported
female friend raped (all female)
16 bullying (14 female, 2 male) 2
sexual assault (all female)
7 domestic violence (all female) 1 rape
(female)
5 physical abuse from parents 1 fear of
stranger “who hurts children” (boy)
(3 male, 2 female) 5 calls – no
data
* Available information analysed from
Survivors Service helpline records
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Primary and Secondary Evaluations:
Telephone calls to survivors helpline – some themes
- Records for 2004/5 showed that
children did not use the helpline. Phone calls were from adults seeking
support for children.
- Some primary 7 children who received
the Tweenees lessons subsequently used the helpline number (applying skills in
real life). Some simply checked out the helpline, others disclosed abuse.
- A significant number of children in
Primary 7 phoned the helpline from other school areas not part of the
evaluation. Given the specificity of the age group, this may be evidence of
children talking to each other across schools leading some children, not
involved in the Tweenees lessons seeking help (generalising knowledge and
skills from child to child?).
- Five pupils from secondary (S1)
phoned the helpline – 4 reported the same incident of rape.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Survivor, Primary and Secondary Interviews
(children’s learning and experiences)
20 interviews (6 survivors, 6 primary
pupils and 8 secondary pupils – matched for gender)
- Nearly all children reported new
learning’s, e.g. what to do in a risky situation, discovering some people
having a tough time and hearing different perspectives
- Little to no negative experiences or
outcomes were reported despite being asked this directly
- Some felt they could talk about
their feelings more openly because they heard others doing so.
- After experiencing the lessons most
reported feeling more confident in keeping themselves safe
- Some judged themselves to be more
assertive in ‘telling’ because of their experience of adults listening,
believing them and giving them more choices within the lessons.
School of Education,
Social Work and Community Education, University of Dundee
Outcomes
Survivor, Primary and Secondary
Interviews
(children’s learning and
experiences)
- Nearly all children affirmed
‘telling’ as the best strategy
- Most felt that following the
lessons, they were more able to keep themselves safe & less likely to be
harmed
- On hearing others stories of harm
most described their reaction as feeling ‘sad and wanted to do something about
it’
- All thought ‘none’ of the ideas
within the lessons were difficult to understand
School of Education,
Social Work and Community Education, University of Dundee
Presenters: Teachers (6)
and 18u Workers (3) – main themes from the interviews
- 18u and teachers held different
goals, the former emphasised disclosure, the latter improving children’s
behaviour towards others.
- 18u workers and teachers held
different definitions of abuse. Teachers tended to operate what could be
described as a ‘narrow’ definition of abuse, in terms of pupils disclosing to
the teacher and requesting that the disclosure be passed on. 18u workers
operated a ‘wide’ definition of abuse incorporating any stories of harm that
were shared.
- Such differences in definition may
have led to the significant differences in the recognition and reporting of
disclosures.
- Teachers thought half the pupils
benefited whereas 18u workers thought all the pupils benefited. This was
underpinned by a difference in the definition of what ‘benefit’ meant.
- Most teachers reported that they
were confident to run the lessons next year.
- Because teachers reported to
experiencing few disclosures during the lessons, none of the teachers felt the
lessons had better prepared them to receive disclosures from children in the
future.
- All thought parents should be more
actively involved.
School of Education,
Social Work and Community Education, University of Dundee
Presenters: Teachers and
18u Workers – main themes from the interviews
- The only detrimental effects on
pupils mentioned was that some boys were showing off during the
lessons.
- The concept seen as easiest for
children to learn was to “say no, go and tell”.
- The concept seen as most difficult
to learn was that “violence is preventable”.
- There were differing views on what
limited the effectiveness of the lessons. Teachers reported on children’s
troublesome behaviour and 18u workers reported on children getting in trouble
at times for their behaviour.
- There were shared views on making
the Tweenees more effective for the future. These included more lessons,
smaller groups and more preparation and training for teachers.
- Another suggestion was to use
multi-agency teams to resource the delivery of the lessons. This was seen as
an opportunity to facilitate closer partnership working between schools and
child protection services.
School of Education,
Social Work and Community Education, University of Dundee
Survivor, Primary and
Secondary Groups: Customer satisfaction – pupils’ views
- All survivors (100%) and most
primary aged pupils liked the Tweenees lessons ‘a lot’ (74%).
- A third of secondary pupils either
‘liked’ or liked the lessons ‘a lot’, with the rest finding the lessons ‘ok’.
A small number of S2 pupils (7) felt bored at times
- The favourite parts of the Tweenees
lessons were the ‘discussions’ (15%) and the games (13%), especially -Truth
Dare Scare , Iffy Jiffy and Traffic lights.
- Pupils liked the writing worksheets
the least (19%)
- Despite being specifically asked,
only 2 primary pupils felt frightened and 3 embarrassed during the lessons. No
pupils reported experiencing upset.
School of Education,
Social Work and Community Education, University of Dundee
Survivor, Primary and
Secondary Groups: Customer satisfaction – pupils’ views
- Almost half (40%) the pupils did not
talk to their parents about the Tweenees lessons. Those that did, spoke about
it being ‘fun’ and ‘learning about safety’.
- ‘How to keep safe’ were the most
frequently reported ‘useful’ ideas (14%) in the lessons.
- Most pupils found the Tweenees
lessons ‘understandable to very understandable’ (over 90%)
- Most pupils felt they had been given
enough information to decide whether to tell if they were being harmed in some
way (over 90%).
- Most reported to feeling good,
comfortable and relaxed during the lessons (80%). A small number felt sad (2%)
on hearing stories of harm.
School of Education,
Social Work and Community Education, University of Dundee
Presenters: Teachers and
18u Workers - Customer satisfaction
Most presenters reported
- Most pupils looked forward to the
lessons
- ‘Truth Dare Scare’ was the favourite
part of the lessons for pupils
- The lessons were
understandable
- No parts of the lessons were
upsetting or anxiety provoking for pupils
- Pupils did not become unduly afraid
of strangers nor overly assertive
- Children of all ages should be
taught such lessons as part of the school curriculum
- They would like to teach these
lessons next year, although one teacher responded ‘no’ in the evaluation form
and ‘yes’ within the later interview.
School of Education,
Social Work and Community Education, University of Dundee
Presenters: Teachers and
18u Workers - Customer satisfaction
There was a difference between 18u
workers and teacher responses in that 18u workers also reported that
children:
- became more assertive through the
‘telling of abuse’
- increased in their frequency in
approaches to them to tell their stories of harm
- applied the strategies they learned
through ‘telling’ within and out-with the lessons
- applied their strategies to other
contexts, e.g. at home or in a small survivors group.
School of Education,
Social Work and Community Education, University of Dundee
Implementation
Integrity:
Video Analysis Summary
From video analysis 18u worker led
delivery of lessons included:
- Open body language and orientation
and movement around the class towards pupils
- High frequency of interactions from
the ‘support’ adult with individual pupils within the class
- Support adult giving individual
pupils an opportunity to tell their story to an affirming adult before giving
the pupil the choice to tell the class.
- Facilitation of ‘peer to peer’ talk
with a high frequency of spontaneous contributions (more akin to the
playground setting)
- Space and time given for longer
turn-taking
- Specific and explicit questions
about harm
- All pupil responses explicitly
received without judgement
- Validation of all viewpoints as
right for that child
- Different pupil responses summarised
to enable pupils to come to their own conclusions
- Disclosures received without
judgement or blocking
- Frequent recognition and reporting
of actual disclosures
- Space and time given for spontaneous
class discipline to occur among pupils rather than active adult
intervention
School of Education,
Social Work and Community Education, University of Dundee
Implementation
Integrity:
the ‘Prevention Mind’
18u worker delivery (from interview)
was underpinned by:
- A high level of motivation and unity
of purpose for the safety and protection of children
- An explicit goal of disclosures as
an outcome
- An expectation that disclosures
would occur
- The motivation and lack of fear to
notice and record disclosures
- A thorough understanding of the
materials and how they work, i.e. the ownership of an active process to
facilitate disclosure
- An absence of ‘adult myths’ about
abuse
- A high level of knowledge about
child development and child protection, e.g. definition, signs and symptoms
and what they mean
- A high knowledge and skill level in
child-centred communication
School of Education,
Social Work and Community Education, University of Dundee
Measure of
Cost-effectiveness
In terms of additional costs the cost
of delivering the Tweenees lessons in the intervention group was worked out per
pupil, per school and per disclosure. This worked out as follows:
- Per pupil = £7.73 (There were
205 pupils in the primary and secondary intervention group. This figure was
divided into the additional total costs to get the cost per pupil for the
intervention group).
- Per disclosure = £21.71 (The
figure selected for disclosures was the 73 disclosures made within the four
lessons in the primary and secondary intervention groups. The additional total
cost was divided by this figure to get a cost per disclosure).
- Per school = £396.25 (There
were four schools in total. This figure was divided into the additional total
costs to get the cost per school).
School of Education,
Social Work and Community Education, University of Dundee
Implications and
Recommendations
- The Tweenees lessons appear to be
one of the most effective abuse prevention programmes to date when delivered
with skill (no other programme has evidenced such levels of disclosure)
- Teachers effectively delivered the
Tweenees lessons to a ‘primary’ prevention level (increased knowledge and
behavioural intention)
- 18u workers effectively delivered
the lessons to both a primary and ‘secondary’ prevention level (disclosure of
abuse)
- Teachers will need focused training
in the ‘Prevention Mind’ in order to deliver the Tweenees materials to a
secondary prevention level.
School of Education,
Social Work and Community Education, University of Dundee
Implications and
Recommendations
- A video should be developed
demonstrating the effective delivery of the Tweenees lessons, incorporating
how to recognise and respond appropriately to disclosure.
- Abuse prevention lessons should be
delivered annually.
- Methods for active parental
involvement should be built into delivery of the lessons.
- A multi-agency model of lesson
delivery should be explored.
- Longer running groups are required
for survivors of abuse.
School of Education,
Social Work and Community Education, University of Dundee
Conclusions
- The Tweenees programme matches other
abuse prevention programmes in achieving on average small knowledge and
behavioural intention gains.
- The Tweenees lessons go
substantially beyond these other programmes by further evidencing the transfer
of these skills into daily life, i.e. children’s use of ‘telling’
behaviour.
- Such developments appear to be
achieved by a highly knowledgeable and experienced prevention mindset coupled
with a child-centred interactive class process.
- Teachers need focused training and
support to develop the knowledge, mindset and child-centred communication
skills that facilitate such a class context and process.
- The most effective and pragmatic way
forward is to draw on the collaborative and joint-expertise of teachers and
child protection workers (survivor services and statutory professionals) in
the delivery of the Tweenees lessons.